Reducing resource consumption of a similarity index in data deduplication

ABSTRACT

Embodiments for reducing resource consumption of a similarity index in data deduplication by a processor. In a similarity index of a deduplication system configured to process snapshots, only a latest generation of repository data is represented in the similarity index where a single latest representative value of an index entry of a snapshot is maintained in the similarity index. Implicit deletion is applied in the similarity index such that the similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected. A subset of bytes of the representative value is maintained in a similarity index entry thereby reducing an input/output (I/O) load on the similarity index of the deduplication system.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates in general to computers, and moreparticularly for reduction of resource consumption of a similarity indexin data deduplication in a computing environment.

Description of the Related Art

In today's society, computer systems are commonplace. Computer systemsmay be found in the workplace, at home, or at school. Computer systemsmay include data storage systems, or disk storage systems, to processand store data. Large amounts of data have to be processed daily and thecurrent trend suggests that these amounts will continue beingever-increasing in the foreseeable future. An efficient way to alleviatethe problem is by using deduplication. The idea underlying adeduplication system is to exploit the fact that large parts of theavailable data are copied again and again, by locating repeated data andstoring only its first occurrence. Subsequent copies are replaced withpointers to the stored occurrence, which significantly reduces thestorage requirements if the data is indeed repetitive.

SUMMARY OF THE DESCRIBED EMBODIMENTS

Embodiments for reducing resource consumption of a similarity index indata deduplication by a processor are provided. In one embodiment, byway of example only, only a latest generation of repository data isrepresented in the similarity index. Implicit deletion is applied in thesimilarity index. A subset of bytes of a full representative value ismaintained in a similarity index entry.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict embodiments of the invention and are not therefore to beconsidered to be limiting of its scope, the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 is a block diagram of an illustration of storage of snapshotswithout data deduplication functionality versus storage of snapshotswith data deduplication functionality;

FIG. 2 is a block diagram illustrating a computing system environmenthaving an example storage device in which aspects of the presentinvention may be realized;

FIG. 3 is a block diagram illustrating a hardware structure of datastorage system in which aspects of the present invention may berealized;

FIG. 4 is an illustration of exemplary tracked blocks enclosed bysimilarity units, in which aspects of the present invention may berealized;

FIG. 5 is an illustration of exemplary pairs of corresponding similarityunits in an input snapshot and a previous snapshot of a data set, hereagain in which aspects of the present invention may be realized;

FIG. 6 is a block diagram of illustrative exemplary functionality forcalculating a representative value based on the digests of the changedtracked blocks of the input similarity unit and the digests of theunchanged tracked blocks of the corresponding similarity unit, again inwhich aspects of the present invention may be implemented;

FIG. 7A is a first portion of a flow chart diagram illustratingexemplary functionality for deduplicating input snapshot data, again inwhich aspects of the present invention may be realized;

FIG. 7B is a second portion of the flow chart diagram first illustratedin FIG. 7A, previously;

FIG. 8 is a block diagram of an exemplary deduplication similarity indexand digest storage segments, again in which aspects of the presentinvention may be implemented;

FIG. 9 is a flow chart diagram illustrating exemplary functionality forcalculating a representative value for a similarity unit in a snapshotdata set, again in which aspects of the present invention may beimplemented;

FIG. 10 is a flow chart diagram illustrating exemplary functionality forcombining matches calculated in multiple deduplication processes toproduce a unified and efficient list of matches, again in which aspectsof the present invention may be implemented;

FIG. 11 is an illustration of exemplary cases in processing of pairs ofdata matches, including truncating, retention and removal of datamatches according to various aspects of the present invention;

FIG. 12A is a first portion of a flow chart diagram illustratingexemplary functionality for deduplicating an initial snapshot in a chainof snapshots, again in which aspects of the present invention may beimplemented;

FIG. 12B is a second portion of the flow chart diagram first illustratedin FIG. 11A previously;

FIG. 13 is a block diagram of an exemplary partition of a representativevalue and the structure of an entry in the similarity index, again inwhich aspects of the present invention may be implemented;

FIG. 14 is a flow chart diagram of an exemplary method for processing oftracked blocks in similarity based deduplication of snapshots data, inaccordance with one embodiment of the present invention;

FIG. 15 is a flow chart diagram of an exemplary method for similaritybased deduplication of snapshots data, in accordance with one embodimentof the present invention;

FIG. 16 is a flow chart diagram of an exemplary method for calculatingrepresentative values for similarity units in deduplication of snapshotsdata, in accordance with one embodiment of the present invention;

FIG. 17 is a flow chart diagram of an exemplary method for combiningdata matches from multiple sources in a deduplication storage system, inaccordance with one embodiment of the present invention;

FIG. 18 is a flow chart diagram of an exemplary method for similaritybased data deduplication of initial snapshots of data sets, inaccordance with one embodiment of the present invention; and

FIG. 19 is a flow chart diagram of an exemplary method for reducingresource consumption of a similarity index in data deduplication, againin accordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

Data deduplication is a highly important and vibrant field in computingstorage systems. Data deduplication refers to the reduction and/orelimination of redundant data. In data deduplication, a data object,which may be a file, a data stream, or some other form of data, ispartitioned into one or more parts called chunks or blocks. In a datadeduplication process, duplicate copies of data are reduced oreliminated, leaving a minimal amount of redundant copies, or a singlecopy of the data, respectively. The goal of a data deduplication systemis to store a single copy of duplicated data, and the challenges inachieving this goal are efficiently finding the duplicate data patternsin a typically large repository, and storing the data patterns in astorage efficient deduplicated form.

A snapshot of data in a data storage system is a copy of the data frozenat a specific point in time. Snapshots are used to obtain consistentcopies of data sets for various use cases, such as backup, archiving,data analysis, development and testing, while allowing applications tocontinue writing and modifying data. Snapshot implementations typicallycreate snapshots in a time that is not dependent on the size of the dataset, and allow applications to write data concurrently with the creationof a snapshot. Typically, after creation of an initial snapshot, thestorage system tracks storage blocks that are changed between creationof consecutive snapshots. For a new further snapshot, the storage systemwill store the storage blocks that have changed since the last snapshot,along with references to unchanged data.

The size of the blocks tracked for changes is usually relatively large,because tracking small blocks in large data repositories requires muchresources and is often prohibitive. Typical sizes of tracked storageblocks can range between 64 KB and 256 KB. However, the changes actuallyapplied to the data can be in a granularity that is smaller than thesize of the tracked storage blocks, especially if the snapshots arecreated frequently. The result is that the size of the data being storedfor a snapshot can be considerably larger than the size of the data thatwas actually changed since the last snapshot.

Adding data deduplication to the creation and storage processes ofsnapshots enables to store only the data that was actually changed,rather than the full sized blocks tracked for changes, thus considerablyreducing storage consumption of the snapshots. FIG. 1 illustrates thisstorage efficiency benefit by showing the marked difference in storeddata sizes, in which the entire contents of the blocks marked asmodified are stored when not using data deduplication, and in which onlythe actual modified sections inside the blocks marked as modified arestored when using data deduplication, as shown in illustration 100.

A series of blocks 102 are shown in sequence. Several blocks 104, 106,and 108 are marked as modified, while the actual modified sections 110,112, and 114 are physically markedly smaller than the physical size ofthe entire blocks marked as modified 104, 106, and 108 and shown. Theblocks 104, 106, and 108 are stored as-is as shown in 120 when nodeduplication (arrow 116) is performed, and their stored data sizeequals to their full size 120. Conversely, the stored data size 122 ismarkedly reduced by use of deduplication (arrow 118), as only the actualmodified sections 110, 112, and 114 are stored as shown.

Since data deduplication can reduce the storage consumption and costsfor storing snapshots, a storage system with data deduplication may beable to store a large number of snapshots, compared to a smallercapacity without data deduplication. There are many applications thatrequire copies of data sets, for example: backup, disaster recovery,instant restore, archive, development, testing, and data analysis.Deduplicated snapshots can enable to provide many of thesefunctionalities directly from a deep store of deduplicated snapshots,rather than having different technologies store their own data copiesseparately. This enables to reduce costs, consolidate resources,simplify management, and increase infrastructure utilization.

To be effective, data deduplication of snapshots should work with asmall granularity. Namely, the size of the deduplication blocks shouldbe typically a few kilobytes. However, in large data repositories thenumber of deduplication blocks of such size can be very large, andstoring deduplication representations for a large number of blocks isoften prohibitive. For example, in a data repository of 1 Petabytes, thenumber of deduplication blocks of 2 Kilobyes is 512 Giga blocks, andmultiplying this by a hash size of 20 bytes (the size of SHA-1), thistotals to 10 Terabytes—required only for the deduplication meta-data.

A new approach is therefore required for data deduplication ofsnapshots, to enable small granularity deduplication while keeping themeta-data size for deduplication low and supporting high efficiency ofthe deduplication process.

The mechanisms of the illustrated embodiments, following, introducevarious aspects of functionality for performing similarity based datadeduplication for snapshots of data, among other aspects, that addressthe need for small granularity deduplication and other needs discussedpreviously.

Similarity based data deduplication, among other benefits, enables theachievement of efficient data deduplication using small data blocks, bypartitioning the search for data matches to two steps. In a first step,similar data is searched in a repository. The granularity for this stepis large data blocks, e.g. 8 MB. In a second step, matches of identicaldata are determined in the similar input and repository data. Thegranularity for this step is smaller data blocks, e.g. 2 KB. Dependingon the matching algorithm that is used, a single byte granularity isalso supported.

The similarity based data deduplication approach is scalable andefficient because the search index used in the similarity step iscompact and simple to maintain and search within. The search index iscompact because, among other benefits, the representation elements usedfor the similarity search are very compact relative to the data theyrepresent, e.g. 16 bytes representing 4 megabytes. Typically, in oneembodiment, the similarity index can fit in memory. Once similar data isfound, digests of the similar data or the similar data itself are loadedinto memory, to perform an efficient data matching process that useshash lookup. This second step process is also efficient in both memoryand processing time. Overall, the similarity based data deduplicationapproach provides scalability to very large data repositories, as wellas high efficiency and performance.

While similarity based approaches provide abundant benefits, thesimilarity based deduplication algorithms provided thus far are designedfor a data backup environment, where large streams of backup data arebeing processed, and each such stream can be partitioned into largechunks of data, e.g. 8 MB, for which similar data can be searched.However, in a data snapshots environment, a data deduplication mechanismis required to deduplicate the tracked storage blocks. These blocks aretypically smaller than the similarity blocks used in the existingsimilarity methods, e.g. 64 KB tracked storage block size versus 8 MBsimilarity block size. In addition, the tracked storage blocks to bededuplicated are typically independent of each other, and can come fromarbitrary locations in the data repository.

In order to deduplicate each of the tracked storage blocks, theresolution of the similarity index should be considerably increasedrelative to the resolution in the backup environment. In theaforementioned example the resolution increase is ×128. In addition, thenumber of IO operations per second for maintaining and searching in thesimilarity index also increases accordingly, and this further grows asthe frequency of the snapshots increases. In typical settings ofsnapshot environments, the required IO operations per second can reachseveral thousands of operations. These high requirements of size and IOoperations per second entailed by the existing similarity baseddeduplication algorithms can result in storage operation inefficienciesin snapshots environments.

Accordingly, new methodologies are beneficial for providing anefficient, scalable, and effective data deduplication solution forsnapshots data. In general, deduplication of snapshots data is anemerging field with a wide applicability scope.

The mechanisms of the illustrated embodiments, in view of the foregoing,introduce various data deduplication functionality that is specificallydesigned and optimized for snapshots data. These mechanisms aim toprovide an efficient, scalable and effective solution for deduplicatingsnapshots data, with a wide applicability scope for a broad range ofstorage systems that include snapshots capabilities.

Returning now to the Figures, and to FIG. 2, specifically, exemplaryarchitecture 200 of a computing system environment is depicted.Architecture 200 may, in one embodiment, be implemented at least as partof a system for effecting mechanisms of the present invention. Thecomputer system embodied in architecture 200 includes at least onecentral processing unit (CPU) 202, which is connected to communicationport 208 and memory device 206. The communication port 208 is incommunication with a communication network 210. The communicationnetwork 210 may be configured to be in communication with systems 212,214 and architecture 200, which may include storage devices 204. Thestorage systems may include hard disk drive (HDD) devices, solid-statedevices (SSD) etc., which may be configured in a redundant array ofindependent disks (RAID).

The operations as described below may be executed on storage device(s)204, located in system 200 or elsewhere, and may have multiple memorydevices 206 working independently and/or in conjunction with other CPUdevices 12. Memory device 206 may include such memory as electricallyerasable programmable read only memory (EEPROM) or a host of relateddevices. Memory device 206 and storage devices 204 are connected to CPU202 via a signal-bearing medium. In addition, memory device 206 and theCPU 202 may be embedded and included in each component of the computingsystem 200. Each storage system may also include separate and/ordistinct memory devices 206 and CPU 202 that work in conjunction or as aseparate memory device 206 and/or CPU 202. As one of ordinary skill inthe art will appreciate, a number of systems 200 may be configured in aconnected and/or distributed way, physically located together or acrossa long geographically connected network, for example.

Turning now to FIG. 3, following; FIG. 3 is an exemplary block diagramshowing a hardware structure of a data storage system 300 that may beused in the overall context of performing functionality according tovarious aspects of the present invention. Data storage system 300 mayfor example, in one embodiment, implement a snapshot-based deduplicationsystem as will be further described.

Host computers 310, 320, and 325, are shown, each acting as a centralprocessing unit for performing data processing as part of a data storagesystem 300. The cluster hosts/nodes (physical or virtual devices), 310,320, and 325 may be one or more new physical devices or logical devicesto accomplish the purposes of the present invention in the data storagesystem 300. Data storage system 300 may implement additionalfunctionality; and one of ordinary skill in the art will recognize thata variety of deduplication, encryption, data processing, etc. hardwareand software, separately or in combination, may be utilized to implementthe functionality according to aspects of the illustrated embodiments.

Network connection 360 may be a fibre channel (FC) fabric, a fibrechannel point-to-point link, a fibre channel over Ethernet (FCoE) fabricor point to point link, a FICON or ESCON I/O interface, any other I/Ointerface type, a wireless network, a wired network, a LAN, a WAN,heterogeneous, homogeneous, public (i.e. the Internet), private, or anycombination thereof. The hosts, 310, 320, and 325 may be local ordistributed among one or more locations and may be equipped with anytype of fabric (or fabric channel) (not shown in FIG. 3) or networkadapter 360 to the storage controller 240, such as Fibre channel, FICON,ESCON, Ethernet, fiber optic, wireless, or coaxial adapters. Datastorage system 300 is accordingly equipped with a suitable fabric (notshown in FIG. 3) or network adaptor 360 to communicate. Data storagesystem 300 is depicted in FIG. 3 comprising storage controllers 340 andcluster hosts 310, 320, and 325. The cluster hosts 310, 320, and 325 mayinclude cluster nodes found in a distributed computing environment.

To facilitate a clearer understanding of the methods described herein,storage controller 340 is shown in FIG. 3 as a single processing unit,including a microprocessor 342, system memory 343 and nonvolatilestorage (“NVS”) 316. It is noted that in some embodiments, storagecontroller 340 is comprised of multiple processing units, each withtheir own processor complex and system memory, and interconnected by adedicated network within data storage system 300. Storage 330 (labeledas 330 a, 330 b, and 330 n herein) may be comprised of one or morestorage devices, such as storage arrays, which are connected to storagecontroller 340 (by a storage network) with one or more cluster hosts310, 320, and 325 connected to each storage controller 340 throughnetwork 360.

In some embodiments, the devices included in storage 330 may beconnected in a loop architecture. Storage controller 340 manages storage330 and facilitates the processing of write and read requests intendedfor storage 330. The system memory 343 of storage controller 340 storesprogram instructions and data, which the processor 342 may access forexecuting functions and method steps of the present invention forexecuting and managing storage 330 as described herein. In oneembodiment, system memory 343 includes, is in association with, or is incommunication with the operation software 350 for performing methods andoperations described herein. As shown in FIG. 3, system memory 343 mayalso include or be in communication with a cache 345 for storage 330,also referred to herein as a “cache memory,” for buffering “write data”and “read data,” which respectively refer to write/read requests andtheir associated data. In one embodiment, cache 345 is allocated in adevice external to system memory 343, yet remains accessible bymicroprocessor 342 and may serve to provide additional security againstdata loss, in addition to carrying out the operations as describedherein.

In some embodiments, cache 345 is implemented with a volatile memory andnonvolatile memory and coupled to microprocessor 342 via a local bus(not shown in FIG. 3) for enhanced performance of data storage system300. The NVS 316 included in data storage controller is accessible bymicroprocessor 342 and serves to provide additional support foroperations and execution of the present invention as described in otherfigures. The NVS 316, may also be referred to as a “persistent” cache,or “cache memory” and is implemented with nonvolatile memory that may ormay not utilize external power to retain data stored therein. The NVSmay be stored in and with the cache 345 for any purposes suited toaccomplish the objectives of the present invention. In some embodiments,a backup power source (not shown in FIG. 3), such as a battery, suppliesNVS 316 with sufficient power to retain the data stored therein in caseof power loss to data storage system 300. In certain embodiments, thecapacity of NVS 216 is less than or equal to the total capacity of cache345.

Storage 330 may be physically comprised of one or more storage devices,such as storage arrays. A storage array is a logical grouping ofindividual storage devices, such as a hard disk. In certain embodiments,storage 330 is comprised of a JBOD (Just a Bunch of Disks) array or aRAID (Redundant Array of Independent Disks) array. A collection ofphysical storage arrays may be further combined to form a rank, whichdissociates the physical storage from the logical configuration. Thestorage space in a rank may be allocated into logical volumes, whichdefine the storage location specified in a write/read request.

In one embodiment, by way of example only, the storage system as shownin FIG. 3 may include a logical volume, or simply “volume,” may havedifferent kinds of allocations. Storage 330 a, 330 b and 330 n are shownas ranks in data storage system 300, and are referred to herein as rank330 a, 330 b and 330 n. Ranks may be local to data storage system 300,or may be located at a physically remote location. In other words, alocal storage controller may connect with a remote storage controllerand manage storage at the remote location. Rank 330 a is shownconfigured with two entire volumes, 334 and 336, as well as one partialvolume 332 a. Rank 330 b is shown with another partial volume 332 b.Thus volume 332 is allocated across ranks 330 a and 330 b. Rank 330 n isshown as being fully allocated to volume 338—that is, rank 330 n refersto the entire physical storage for volume 338. From the above examples,it will be appreciated that a rank may be configured to include one ormore partial and/or entire volumes. Volumes and ranks may further bedivided into so-called “tracks,” which represent a fixed block ofstorage. A track is therefore associated with a given volume and may begiven a given rank.

The storage controller 340 may include a data deduplication module 355,a similarity module 357, and a similarity search module 359, as well asa number of repositories (not shown). The data deduplication module 355,similarity module 357, similarity search module 359, and repositoriesmay operate in conjunction with each and every component of the storagecontroller 340, the hosts 310, 320, 325, and storage devices 330. Thedata deduplication module 355, similarity module 357, similarity searchmodule 359, and repositories may be structurally one complete module ormay be associated and/or included with other individual modules or otherstructures. The data deduplication module 355, similarity module 357,similarity search module 359, and repositories may also be located inthe cache 345 or other components, as one of ordinary skill in the artwill appreciate.

The data deduplication module 355, similarity module 357, similaritysearch module 359, and repositories may individually and/or collectivelyperform various aspects of the present invention as will be furtherdescribed. For example, the data deduplication module 355 may performvarious data deduplication functionality in accordance with aspects ofthe illustrated embodiments.

The similarity module 357 may perform a variety of functionality as willbe further described, such as partitioning input snapshot data intochanged tracked blocks, grouping the changed tracked blocks intoenclosing similarity units, and partitioning the changed tracked blocksinto variable-sized deduplication blocks. The similarity search module359 may also perform a variety of functionality as will be furtherdescribed, such as examining deduplication coverage and conductingsimilarity searches between input similarity units and correspondingsimilarity units of a previous snapshot. As one of ordinary skill in theart will appreciate, the data deduplication module 355, similaritymodule 357, similarity search module 359, and repositories may make uponly a subset of various functional and/or functionally responsibleentities in the data storage system 300.

Other ancillary hardware may be associated with the storage system 300.For example, as shown, the storage controller 340 includes a controlswitch 341 for controlling the fiber channel protocol to the hostcomputers 310, 320, 325, a microprocessor 342 for controlling all thestorage controller 340, a nonvolatile control memory 343 for storing amicroprogram (operation software) 350 for controlling the operation ofstorage controller 340, data for control, cache 345 for temporarilystoring (buffering) data, and buffers 244 for assisting the cache 245 toread and write data, a control switch 341 for controlling a protocol tocontrol data transfer to or from the storage devices 330, the datadeduplication module 355, similarity module 357, similarity searchmodule 359, and repositories, or other blocks of functionality, in whichinformation may be set. Multiple buffers 344 may be implemented with thepresent invention to assist with the operations as described herein.

In one embodiment of the present invention, the input changed trackedblocks (i.e., tracked blocks that were determined as changed by thesnapshot mechanism) are grouped by the mechanisms of the illustratedembodiments by enclosing larger “similarity units,” and are thenprocessed for deduplication according to these similarity unit groups.

Turning to FIG. 4, following, an illustration 400 of this functionalityis shown, with a group of input changed tracked blocks 402 that areshown as part of a larger sequence of blocks 408. The changed trackedblocks 402 are organized by an enclosing similarity unit 404 andprocessed for deduplication as shown. The changed tracked blocks 402have an exemplary size of 64 KB, and are enclosed by a similarity unit404 having an exemplary size of 8 MB. Each one of the changed trackedblocks is processed into a number of deduplication blocks 406 having anexemplary size of 2 KB.

In one embodiment, the boundaries of the similarity units 404 arealigned to the size of the tracked blocks 402, namely a tracked blockcan not span over similarity units. The start positions of thesimilarity units are aligned to their size. For example: Assuming thatthe size of a similarity unit is 8 MB and the size of a tracked block is64 KB, then the start positions of the similarity units are 0, 8 MB, 16MB, 24 MB, etc. Consider for example 4 tracked blocks, starting atpositions 896 KB, 5120 KB, 21824 KB, and 27840 KB. The first two trackedblocks will be grouped within the similarity unit starting at position0. The third tracked block will be grouped within the similarity unitstarting at position 16 MB. The fourth tracked block will be groupedwithin the similarity unit starting at position 24 MB.

Only similarity units that contain at least one input changed trackedblock are processed for deduplication. The input tracked blocks in agroup are processed together for deduplication as shown. Further, and asshown, each input changed tracked block 402 is further partitioned intovariable sized deduplication blocks, e.g. of mean size 2 KB. The set ofdeduplication blocks starts at position 0 of each changed tracked block,and deduplication blocks do not span over tracked blocks.

The model depicted in FIG. 4 enables to achieve high resolutiondeduplication for snapshots while keeping the similarity index compact,and performing similarity search with a lower resolution than that ofthe tracked blocks, which considerably reduces the Input/Output (IO)operations per second.

An observation underlying another aspect of the illustrative embodimentsis that using prior knowledge for attempting to perform efficientdeduplication of a group of input changed tracked blocks is moreefficient, and only if this attempt does not produce sufficientdeduplication a global similarity search is performed for the inputgroup of blocks. The mechanisms of the illustrated embodiments firstidentify a similarity unit in the previous snapshot (if exists) thatcorresponds to the input similarity unit, and attempt deduplication withthis corresponding similarity unit.

A corresponding similarity unit in a previous snapshot is defined ashaving a beginning position in the previous snapshot, which is identicalto the beginning position of the input similarity unit in the inputsnapshot. FIG. 5, following, in illustration 500, shows pairs ofcorresponding similarity units in an input snapshot (e.g., inputsnapshot 502) and a previous snapshot (e.g., previous correspondingsnapshot 504) of a data set.

Referring to illustration 500, in one embodiment, the mechanisms of thepresent invention load the deduplication digests of the correspondingsimilarity unit of the previous snapshot into memory, and apply adigests matching process to the digests of the input and thecorresponding similarity units. The mechanisms then check the coverageof deduplication. If coverage is sufficient, (e.g., above or equal to athreshold), then the calculated matches are used and processing proceedsto the next input similarity unit. If coverage is insufficient, (e.g.,below a threshold), then the mechanisms apply a similarity search forthe input similarity unit, by searching for a similar data unit in asimilarity index, and if found the mechanisms load the deduplicationdigests of the found similarity unit into memory, apply a digestsmatching process to the digests of the input and the found similarityunits, and combine the matches found in the current and the previousmatching processes to a unified set of matches.

With the above model, the following challenge becomes apparent: For eachinput similarity unit a representative value has to be calculated toenable searching for similar data in a similarity index and to enablestoring the information of the input similarity unit in the similarityindex. To be effective, a representative value has to be calculatedbased on the full data of a similarity unit. However, in the above modelonly the data of the changed tracked blocks is provided in the input,and reading the unchanged data is inefficient and adds considerableoverhead.

To address the challenge previously described, the mechanisms of theillustrated embodiments introduce an additional aspect, which is a novelmethodology for calculating representative values for input similarityunits in snapshots data. In the mechanisms of the illustratedembodiments, for an input similarity unit, the digests of acorresponding similarity unit in a previous snapshot of the data set(i.e. the corresponding similarity unit has a position in the previoussnapshot which is the same as the position of the input similarity unitin the input snapshot), are loaded in memory for applying deduplicationwith the input similarity unit. Subsequently, the mechanisms combine thedigests of the unchanged tracked blocks sourced from the digests of thecorresponding similarity unit (which are loaded in memory) with theinput digests of the changed tracked blocks of the input similarityunit, to produce a representative value for the input similarity unit.

The calculation of a representative value for an input similarity unitbased on the digests of the changed tracked blocks of the inputsimilarity unit and the digests of the unchanged tracked blocks of thecorresponding similarity unit is depicted in an exemplary embodiment asillustration 600 in FIG. 6, following. An input similarity unit 602 anda corresponding similarity unit 604 are shown. For the input similarityunit 602, the digest values of the changed tracked blocks 606 areincluded in the calculation of the representative value 610 for theinput similarity unit 602. In addition, the digest values of theunchanged tracked blocks 608 sourced from the digest values of thecorresponding similarity unit 604 are also included in the calculationof the representative value 610 for the input similarity unit 602. Thedigest values of the input changed tracked blocks 606 and the digestvalues of the unchanged corresponding tracked blocks 608 are combined ina calculation of the representative value 610 for the input similarityunit 602 as shown. This representative value is thus calculated based onthe entire data of the input similarity unit 602. Since the digestsrequired for this calculation are loaded into memory for thededuplication process itself, there is no overhead for this calculation.

An additional aspect of the illustrated embodiments is a novelmethodology for combining data matches calculated in multiplededuplication processes for an input similarity unit, to provide anefficient set of data matches covering the input data. This methodologyis required to combine the data matches generated by deduplication witha corresponding similarity unit, and the data matches generated bydeduplication with a similarity unit found using the similarity index.The methodology is generic for combining data matches generated byvarious deduplication processes performed on different repository data.

In view of the preceding, in one embodiment, the data matches are sortedby their start positions, and subsequently, data matches that are fullyenclosed within other data matches or are too small for citing areeliminated. Next, each pair of data matches with an overlap section isscanned in ascending order, and from each pair a data match is selectedto be retained in its full size. One example of a selection criterion isto select the data match whose size is maximal. Next, the other datamatch in the pair is truncated, by removing the overlap section from theother data match. Then the truncated data match is retained if it is notfully enclosed within the next data match in the sorted list of datamatches and if it is not too small for citing. Otherwise, the truncateddata match is removed. Then, the next pair of data matches with anoverlap section is process by the methodology.

An additional aspect of the mechanisms of the illustrated embodimentsconcerns a methodology for efficient deduplication of an initialsnapshot in a chain of snapshots, namely a snapshot that does not have apreceding snapshot of the same data set. The problem addressed is thatfor an initial snapshot there is no previous snapshot to provide anefficient alternative for finding similar data. Accessing the globalsimilarity index for each of the similarity units of the initialsnapshot can consume much resources. Moreover, initial snapshots containthe full data of their data sets, and therefore can be large, i.e.,consisting of a large number of tracked blocks and similarity units. Theobjective of the proposed methodology is to reduce accesses to theglobal similarity index.

In the mechanisms of the illustrated embodiments, a separate, in-memorysimilarity index may be built during the deduplication process of theinitial snapshot. This index contains representations of only the dataof the initial snapshot, and is therefore small and enables efficientaccess. The index may be denoted as “intra-snapshot similarity index”.This model leverages the higher frequency of similar data inside datasets. Also note that for an initial snapshot, the full data of thesnapshot is provided in the input, thus the mechanisms of theillustrated embodiments are able to calculate a representative value foreach of the input similarity units.

In one embodiment, for each similarity unit, first the intra-snapshotsimilarity index is searched for similar data, and only if no similardata is found or deduplication with the found similar data isinsufficient—the embodiment proceeds to search in the global similarityindex. When processing of the entire initial snapshot is complete, theintra-snapshot similarity index is integrated into the global similarityindex. This is done using an efficient method; for example if theindexes are search trees, then the tree of the intra-snapshot index canbe merged structurally into the tree of the global similarity index, orits entries can be bulk inserted into the global similarity index,depending on the type of the tree.

An additional optimization in this context is that the intra-snapshotsimilarity index can be built using a resolution that is higher than theresolution of the the global similarity index. Namely, the similarityunits used to build and to search within the intra-snapshot similarityindex can be smaller than the similarity units used for the globalindex, thus providing higher resolution similarity search within asnapshot. This is enabled because the intra-snapshot similarity index issmaller typically in orders of magnitude relative to the globalsimilarity index. This is further enabled, as the mechanisms of theillustrated embodiments allow for representative values of a lowerresolution to be efficiently calculated from representative values of ahigher resolution. Namely, the representative values of theintra-snapshot similarity index are efficiently combined to producerepresentative values for searching in and later inserting into theglobal similarity index.

An additional aspect of the mechanisms of the illustrated embodimentsconcerns a set of techniques that enable to further reduce the IOconsumption and size of the similarity index. The challenge beingaddressed is that for large data repositories or repositories thataccommodate frequent snapshots, the IO consumption required for thesearch and update operations on the similarity index can be high. Theinventive techniques to address this challenge are outlined, following.

A first technique provides that the similarity index represents only alatest generation of the repository data. Namely, for eachrepresentative value, a single latest entry is maintained.

A second technique provides for implicit deletion in the similarityindex. The problem addressed is that deletion is not limited and doesnot depend on the rate of incoming data, and can therefore create a highload of IO operations on the similarity index. Based on experience fromcurrent deduplication mechanisms, explicit deletion is typically ahighly resource consuming operation. In the mechanisms of theillustrated embodiments, entries in the similarity index becomedesignated to be removed when a snapshot is deleted, and when a newsnapshot is processed and its new representative values replace existingrepresentative values (which become designated to be removed). Theseoperations do not perform explicit deletion in the present invention,and entries designated to be removed by these operations are then eitherupdated by new snapshots (thus reducing IO consumption) or identifiedand removed within search operations. The details of these techniquesare provided further, following.

A third technique provides that while, for example, the 20 bytes (ormore) of the cryptographic hash value that constitutes the digest value,only 8 bytes are stored in the similarity index. In one embodiment, asimilarity index entry consists of 8 bytes that are a subset of thedigest value of a representative value, and 8 bytes that are a storagereference to the digests of the similarity unit to which the currentrepresentative value corresponds to. The 8 byte space provides asufficient distribution and uniqueness for the representative values,and the 12 remaining bytes are stored within the digests of a similarityunit and are verified within a search operation to validate that aninput and a repository representative values indeed match.

Now that some context for various aspects of the illustrated embodimentshave been introduced, as well as challenges and inventive techniques andmethodologies to address them, the following describes the functionalityinherent in the mechanisms of the illustrated embodiments for providingdeduplication functionality for input snapshot data. This functionalityis also illustrated by the method 700 shown in FIGS. 7A and 7B, whichdepicts an exemplary method 700 for providing deduplicationfunctionality for input snapshot data, in which aspects of the presentinvention may be implemented.

Method 700 in FIG. 7A begins with the input of a set of changed trackedblocks, where the size of each of the changed tracked blocks is, forexample, 64 KB, and the blocks' positions in the storage are arbitraryand depend on the modification of data in the storage between the timeof the previous snapshot and the time of the current snapshot. This isdescribed and depicted in step 702 as shown.

In a subsequent step 704, the data is partitioned into similarity unitsof a sufficiently large size, (e.g., 8 Megabytes). A sufficiently largesize is such that representing all the similarity units in a repositoryrequires a storage size that is sufficiently small. Each similarity unitmay enclose zero or more input changed tracked blocks. In decision step706, method 700 queries whether there is an additional input similarityunit that encloses one or more input changed tracked blocks. If no, themethod 700 then ends (step 728). If so, for each input similarity unitthat encloses one or more input changed tracked blocks, deduplicationblocks (e.g., of size 2 KB), are calculated for the input changedtracked blocks within the input similarity unit, and a digest value,e.g. SHA-1 value, is calculated for each deduplication block (step 708).

In a subsequent step, a corresponding similarity unit within theprevious corresponding snapshot is identified. The correspondingsimilarity unit has a nominal start position in the previouscorresponding snapshot that is the same as the nominal start position ofthe input similarity unit in the input snapshot. The digest values ofthe deduplication blocks of the identified corresponding similarity unitare loaded into memory (step 710).

The digest values of the input similarity unit and the digest values ofthe corresponding similarity unit are matched to find identical datasections (step 712).

Method 700 then moves to decision step 714, which queries whether thededuplication coverage of the input similarity unit is less than athreshold. If the coverage is not less than a threshold, the method 700returns to step 706 as shown. If the coverage is indeed less than athreshold, then the method 700 continues to process the current inputsimilarity unit as follows.

A representative value is calculated for the input similarity unit basedon the digests of the input similarity unit and the digests of thecorresponding similarity unit of the previous snapshot (step 716).Specifically, the digests of the unchanged tracked blocks sourced fromthe corresponding similarity unit are considered together with thedigests of the changed tracked blocks of the input similarity unit, tocalculate the representative value.

The calculated representative value is searched in a similarity index ofrepresentative values. (step 718). Turning now to FIG. 7B, whichcontinues the exemplary method previously began in FIG. 7A, previously,if a matching representative value is not found in the similarity index(decision step 720), then the method 700 proceeds to process the nextinput similarity unit (returning to step 706 as previously). If amatching representative value is found, then the method 700 continues toprocess the current input similarity unit as follows.

The digests of the found similarity unit referenced by the foundrepresentative value are loaded into memory (step 722). The digestvalues of the input similarity unit and the digest values of the foundsimilarity unit are matched to find identical data sections (step 724).Finally, the identical sections calculated based on the correspondingsimilarity unit and the found similarity unit are combined to produce afinal list of identical sections (step 726). The method 700 thenproceeds to process the next input similarity unit by returning to step706 as previously.

Exemplary methodologies for processing a single input similarity unitare now described. In one embodiment, deduplication blocks, (e.g., ofsize 2 KB), are calculated for each one of the changed tracked blockswithin the input similarity unit. The boundaries of the deduplicationblocks can be calculated using a contents defined chunking method, forexample, that provides variable sized deduplication blocks. For eachdeduplication block a cryptographic hash value, e.g. a SHA-1 value,named a digest value, is calculated. The information of a digestincludes the digest value and the position and size of the correspondingdeduplication block.

The digests of a similarity unit are stored in a continuous storagesegment, where their order in the storage segment corresponds to theirorder of appearance in the data. Each storage segment corresponds to aspecific similarity unit. A representative value entry in the similarityindex contains a storage reference that enables to efficiency identifythe location in the storage of the digest storage segment correspondingto the similarity unit represented by the entry.

FIG. 8, following, in illustration 800, depicts the metadata layout. Anexemplary similarity index 802 referencing representative storagesegments of digests 804 that are referenced by entries in the similarityindex as shown. Each similarity index entry 806 includes a correspondingrepresentative value as previously described that was calculated, and astorage reference to a segment of digests that enables to efficiencyidentify the location in the storage of the digest storage segmentcorresponding to the similarity unit represented by the entry.

Accordingly, in one embodiment, each storage segment has a metadatasection that includes the following components: (1) CRC; (2) theposition in the storage segment of the representative digest of thesimilarity unit corresponding to the storage segment; (3) theidentification of the latest snapshot that updated the storage segment;and (4) a map of nominal base positions for sub-sections of the storagesegment, which enables efficient position calculation of a specificdigest in a segment (as the digest blocks are of variable size).

In the mechanisms of the illustrated embodiments, a digests storagesegment has a fixed physical size and a fixed nominal size. Examples ofthese sizes may be a 48 KB physical size and an 8 MB nominal size. Thefixed physical and nominal sizes enable O(1) addressing of digestsstorage segments. Namely, given an input nominal position, the physicalposition of the digests storage segment covering the input nominalposition is known in O(1). The physical and nominal sizes of a digestsstorage segment are calculated to provide a minimal probability ofexcess digests per storage segment and be IO efficient. Representativedigests for similarity units are calculated based on the maximal numberof digests for a storage segment, thus avoiding excess digests that arenot stored. This is a repeatable calculation method that is optimizedfor the above design of digests storage segments.

Input and repository digests may be matched using a digests matchingalgorithm. This algorithm, for example, loads one set of digests into ahash table, and searches using the hash table for each of the digests ofthe other set. For each match found, the match is attempted to beextended with consecutive matching digests. A pair of digests aredetermined as matching when their cryptographic hash values and theirdeduplication block sizes match.

Turning now to FIG. 9, following, an exemplary method 900 forcalculating a representative value for an input similarity unit in aninput snapshot is shown, in which aspects of the present invention maybe implemented. As an introductory step, the input for the method 900 isa set of changed tracked blocks of an input similarity unit (step 902).Note that only the data of the input changed tracked blocks is availablein the input, and the data of the unchanged tracked blocks is notavailable in the input.

In step 904, the maximal digest value is calculated for the inputchanged tracked blocks. This is done during the calculation of thedigest values for the input changed tracked blocks (these digest valuesare required for deduplication on the input data). In a subsequent step906, the maximal digest value for the unchanged tracked blocks iscalculated. This is done based on the digest values of the unchangedtracked blocks. These digest values are sourced from the correspondingsimilarity unit in the previous snapshot of the data set. In theexemplary method 900, these digest values are already loaded into memoryto facilitate deduplication with the corresponding similarity unit.

Additional optimizations are provided to the method 900, as follows, inone embodiment. If the maximal digest value is stored for each trackedblock, then rather than using the digests these stored maximal digestvalues may be used to calculate the maximal digest value for theunchanged tracked blocks. Additionally, if the maximal digest value isstored for each similarity unit (rather than for each tracked block),then if the input changed tracked blocks do not overwrite the positionof the stored maximal digest value then the stored maximal digest valueis valid to be used in the calculation as the current maximal digestvalue for the unchanged tracked blocks. Only in the case where the inputchanged tracked blocks overwrite the position of the stored maximaldigest value and the maximal digest value of the input changed trackedblocks is smaller than the stored maximal digest value, then a maximaldigest value is calculated for the unchanged tracked blocks.

In a subsequent step 908, the maximal of these two digest values iscalculated, and the deduplication block associated with the maximaldigest value is identified. This deduplication block can be either inthe changed tracked blocks or in the unchanged tracked blocks.

A repeatable position shift, (e.g., one position next), is applied forthat deduplication block to identify a representative deduplicationblock (step 910). Finally, the digest value of the representativededuplication block is selected to be the representative value of theinput similarity unit (step 912). The method 900 ends (step 914).

FIG. 10, following, illustrates an exemplary method 1000 for combiningdata matches calculated in multiple deduplication processes for an inputsimilarity unit, to produce a unified and efficient list of datamatches, in which aspects of the present invention may be implemented.In the depicted embodiment, a ‘data match’ is defined as a sequence ofmatching bytes in the input and reference data.

The input is a list of data matches calculated by multiple deduplicationprocesses for an input similarity unit, where the data matches may beoverlapping and unordered (step 1002). The data matches are sorted bytheir start positions in ascending order (step 1004). Data matches thatare completely enclosed within other matches, or are determined to betoo small for citing, are removed (step 1006). Each pair of the datamatches having an overlap section is then scanned in an ascending orderof position (step 1008).

Continuing to decision step 1010, which queries whether an additionalpair of data matches having an overlap section is available (if no, themethod 1000 ends in step 1020), for each such pair: (1) a data match isselected from the pair to retain in its full size; this data match isdenoted as ‘data match A’ (step 1012); (2) the other data match, denotedas ‘data match B’, is truncated by removing the overlap section fromdata match B so that it will not overlap with data match A (step 1014);this truncation process may cause the truncated data match B to becomefully enclosed within the next data match in the sorted list of datamatches; (3) if the truncated data match B is fully enclosed within thenext data match in the sorted list of data matches, or if the truncateddata match B is determined to be too small for citing, then thetruncated data match B is removed; otherwise the truncated data match Bis retained (step 1016). The method 1000 then moves to process the nextpair of data matches having an overlap section (returning to decisionstep 1010 as previously).

Selection of the data match to retain in its full size from a pair ofdata matches may be based on a variety of criteria. In one embodiment,the selection of the data match may be determined by which of the pairhas a maximal size. In another embodiment, the selection may bedetermined by the data match whose position in the reference data isclosest to the positions of any previous selected data matches. In stillanother embodiment, the selection may be determined by a combination ofthe previous criteria. Any other selection criteria may also be used ina data match determination.

FIG. 11, following, graphically illustrates the processing performed foreach pair of data matches, including truncation, retention and removaloperations described in FIG. 10, previously, as shown in three examplecases. In the depicted ‘Case A,’ in the processing of data matches 1 and2, data match 2 is larger and is retained in its full size, while datamatch 1 is truncated with the overlap section, and its truncated form isretained. The next pair to process is data matches 2 and 3 as shown.

In the depicted ‘Case B,’ in the processing of data matches 1 and 2,data match 1 is determined to be larger and is retained in its fullsize, while data match 2 is truncated with the overlap section. Thetruncated form of data match 2 is fully enclosed within data match 3;therefore, data match 2 is removed, and the next pair to process is datamatches 1 and 3 as shown.

Finally, in the depicted ‘Case C,’ in the processing of data matches 1and 2, data match 1 is determined to be larger and is retained in itsfull size, while data match 2 is truncated with the overlap section. Thetruncated form of data match 2 is not fully enclosed within data match3; therefore, the next pair to process is data matches 2 (truncated) and3 as shown.

FIGS. 12A and 12B, following, depict an exemplary method 1200 fordeduplicating an initial snapshot in a chain of snapshots according toone embodiment of the present invention, namely a snapshot that does nothave a preceding snapshot of the same data set. Turning first to FIG.12A, in step 1202, the input to the method 1200 is an initial snapshotin a chain of snapshots. The data of the input initial snapshot is thenpartitioned into similarity units (e.g., of size 8 MB) (step 1204).

Method 1200 then moves to decision step 1206, which queries whether anadditional input similarity unit is available in the input initialsnapshot. In other words, for each input similarity unit, the method1200 moves to step 1208 as shown, to calculate deduplication blocks(e.g., of size 2 KB) for the tracked blocks of the input similarityunit. In addition, a digest value, (e.g., SHA-1 value), is calculatedfor each deduplication block. If an additional input similarity unit isnot found, the method 1200 moves to step 1226 in FIG. 12B, as will befurther described.

Continuing to step 1210, high resolution representative values arecalculated for the similarity unit (e.g., each representative valuecorresponds to a 2 MB sub-unit). Each one of the high resolutionrepresentative values is searched in an intra-snapshot similarity index,and similar sub-units are identified (step 1212). If similar sub-unitswere found, then the digests of the input similarity unit and thedigests of the found sub-units are matched to find identical datasections (step 1214).

The method 1200 then proceeds to decision step 1116, which querieswhether the deduplication coverage of the input similarity unit is lowerthan a threshold. If so, then the method 1200 proceeds to steps 1218,1220, 1222 and 1224 as follows. If not, the method 1200 returns todecision step 1206 as shown.

Returning to decision step 1216, and following the progression of method1200 through the assumption that the examined deduplication coverage ofthe input similarity unit is lower than a threshold, a representativevalue is calculated for the input similarity unit based on the highresolution representative values of the sub-units in a subsequent step1218. The calculated representative value is then searched in a globalsimilarity index (step 1220).

Turning now to FIG. 12B, and continuing with step 1222, if a similarunit was found, then the digests of the input similarity unit and thedigests of the found similarity unit are matched to find identical datasections. The identical sections calculated based on the sub-units foundin the intra-snapshot similarity index and the similarity unit found inthe global similarity index are combined to produce a final list ofidentical sections (step 1224).

Returning now to decision step 1206 in FIG. 12A, and assuming that noadditional input similarity unit is available in the input initialsnapshot (in other words, when processing of all the input similarityunits of the input initial snapshot has been completed), the method 1200moves to step 1226 as follows. The intra-snapshot similarity index isintegrated into the global similarity index, e.g. by tree merge or bulkinsertion of the intra-snapshot index entries into the global similarityindex. The method 1200 then ends (step 1228).

Several inventive aspects may be applied as part of the mechanisms ofthe illustrated embodiments to further reduce the IO consumption andsize of the similarity index. In one embodiment, each entry in thesimilarity index may be configured with e.g. 8 bytes that are a subsetof the representative digest value of the corresponding similarity unit,and e.g. 8 bytes that are a storage reference to the digests segment ofthe corresponding similarity unit. The former 8 bytes are a subset ofthe e.g. 20 bytes (or more) of the cryptographic hash value thatconstitutes the representative digest value.

The embodiment just described is depicted as illustration 1300 in FIG.13, following, to show an exemplary partition of a representative valueand the structure of an entry in the similarity index. A representativevalue of, for example 20 bytes, incorporates a portion used as therepresentative value in the similarity index, of size for example 8bytes (portion 1304), and a remaining portion that is stored with thedigests of the similarity unit, of size for example 12 bytes (portion1306). The 8-byte space in portion 1304 provides a sufficientdistribution and uniqueness for the representative values, and the 12remaining bytes in portion 1306 are stored in the representative digestrecord which is stored in the digests segment of the corresponding(referenced) similarity unit. These 12 bytes are verified within asearch operation to validate that an input and a repositoryrepresentative values indeed match. Namely, when a match is found in thesimilarity index (based on the 8 bytes subset), the remaining 12 bytesof the representative value are obtained and verified to match the inputrepresentative value. Similarity index entry 1308, e.g. of size 16bytes, then includes a portion 1310 storing a representative value, ofsize for example 8 bytes, and a portion 1312 used to store a storageaddress of the corresponding digests segment, of size for example 8bytes.

In another inventive aspect, in one embodiment, the similarity index isconfigured to represent only a latest generation of the repository data.Namely, for each representative value, a single latest entry ismaintained. Specifically, when a new representative value has a valuewhich is equal to that of an existing representative value, the newvalue overwrites the existing value.

In an additional inventive aspect, in one embodiment, implicit deletionis applied in the similarity index. In the mechanisms of the illustratedembodiments, explicit deletion operations are not performed, and entriesdesignated to be removed are either updated by new snapshots (thusreducing IO consumption) or identified and removed within searchoperations. There are two cases, specified next as exemplaryembodiments, where entries in the similarity index are designated to beremoved.

In the first case, a new snapshot is processed and its newrepresentative values replace existing representative values. A newsnapshot adds only the representative values of the changed similarityunits in the snapshot. The representative values of the unchangedsimilarity units remain as is. Consider a changed similarity unit thatcontributes a representative value into the similarity index. If the newrepresentative value of the similarity unit is equal to its previousrepresentative value (i.e., the change in the data did not cause changein the representative value), then the new representative value willreplace, and thus also implicitly delete, the previous representativevalue. If the new representative value of the similarity unit isdifferent from its previous representative value, then the previousrepresentative value of that similarity unit remains in the index and isdesignated to be removed. The methods for identifying and processingsuch representative values are specified in the following.

In the second case, an existing snapshot is deleted. Explicit deletionof entries is not performed, and therefore the representative values ofa deleted snapshot remain in the index as designated to be removed, andthe methods for identifying and processing such representative valuesare specified in the following. Entries in the similarity index thatbecome designated to be removed by any of the above processes (i.e., newsnapshot processing, deletion of a snapshot) are then processed by oneof two exemplary processes.

In a first exemplary process, an entry designated to be removed may beupdated directly by a new entry that has the same representative value(but references different digests). In this way the entry designated tobe removed is overwritten (and thus removed) by the new entry.

In a second exemplary process, an entry designated to be removed may beprocessed by a search operation of a given representative value.Processing of such entries is classified into the following cases.

In the first case the storage reference to a digests segment is nolonger valid. In this case the invalid storage reference will beidentified as such by the search operation, and the entry will beremoved within the search operation.

In the second case the storage reference to a digests segment is reusedby another similarity unit and is therefore valid, but therepresentative value of that similarity unit is different from therepresentative value designated to be removed. In this case, the searchoperation checks if the two representative values match (checking theirfull size, e.g., 20 bytes), and because they are different, the searchoperation will identify the entry designated to be removed, and willremove that entry.

In the third case the storage reference to a digests segment is reusedby another similarity unit and is therefore valid, and therepresentative value of that similarity unit is equal to therepresentative value designated to be removed. In this case the newrepresentative value directly updates (overwrites) the existingrepresentative value, and therefore this case is equivalent to the firstcase above.

The functionality imparted by implicit deletion in the illustratedembodiments serves to reduce IO operations by embedding removal ofentries designated for removal within update operations and searchoperations. With explicit deletion the cost for removing entries is paidupfront, while with implicit deletion as suggested herein, the cost isdeferred and only a portion of that cost is eventually applied (forremoval of entries that were processed by a search operation beforebeing processed by an update operation). Furthermore, entries designatedfor removal that exist in the index are safe in terms of deduplication.

Turning now to FIG. 14, method 1400 illustrates an exemplary method forprocessing tracked blocks in a data storage implemented with datadeduplication by a processor, in which various aspects of the presentinvention may be implemented. Method 1400 begins (step 1402) with thepartitioning of input snapshot data into changed tracked blocks (step1404). The changed tracked blocks are then grouped into enclosingsimilarity units (step 1406). The changed tracked blocks are thenpartitioned into variable-sized deduplication blocks (step 1408).Finally, those similarity units that contain at least one input changedtracked block are processed for deduplication (step 1410). The method1400 then ends (step 1412).

FIG. 15, following, illustrates an exemplary method 1500 fordeduplication of an input snapshot in a data storage by a processor, inwhich aspects of the present invention may be implemented. The method1500 begins (step 1502). An input snapshot data is partitioned intosimilarity units 1504. The input similarity unit is deduplicated with acorresponding similarity unit of a previous snapshot (step 1506).Deduplication coverage is examined (step 1508). If a deduplicationcoverage threshold is not met, a similarity search is conducted and theinput similarity unit is deduplicated with a found similarity unit (step1510). The method 1500 then ends (step 1512).

FIG. 16, following, illustrates an exemplary method 1600 for calculatinga representative value for an input similarity unit in datadeduplication of snapshots data by a processor, again in which variousaspects of the present invention may be implemented. Method 1600 begins(step 1602) with the identification of a corresponding similarity unitof a previous snapshot that corresponds to an input similarity unit(step 1604). A calculation is performed based on digests of the inputsimilarity unit and digests of the corresponding similarity unit (step1606). Based on the calculation, a representative value is produced forthe input similarity unit (step 1608). The method 1600 then ends (step1610).

FIG. 17, following, illustrates an exemplary method 1700 for combininginput data matches from multiple sources in deduplication of data indata storage by a processor, according to one embodiment the presentinvention. Method 1700 begins (step 1702) with the calculation of inputdata matches using a plurality of deduplication processes referencing aplurality of repository data segments for the input data (step 1704). Acombined list of output data matches, is then calculated (step 1706).The method 1700 then ends (step 1708).

Turning now to FIG. 18, an exemplary method 1800 for data deduplicationof an initial snapshot of a data set in a storage system by a processor,is depicted. Method 1800 begins (step 1802) with the building of anintra-snapshot similarity index, inclusive of representations of thedata inside the initial snapshot (step 1804). The intra-snapshotsimilarity index is then used for deduplication of the initial snapshot(step 1806). The intra-snapshot similarity index is then merged with aglobal similarity index when processing of the initial snapshot iscomplete (step 1808). The method 1800 then ends (step 1810).

FIG. 19, following, illustrates an exemplary method 1900 for reducingresource consumption of a similarity index in data deduplication by aprocessor according to one embodiment of the present invention. Method1900 begins (step 1902), by representing in the similarity index only alatest generation of repository data (step 1904). Implicit deletion isapplied in the similarity index (step 1906). A subset of bytes of a fullrepresentative value is maintained in a similarity index entry (step1908). The method 1900 then ends (step 1910).

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A method for reducing resource consumption of asimilarity index in data deduplication by a processor, comprising:partitioning input snapshot data into changed tracked blocks; groupingthe changed tracked blocks into enclosing similarity units; processingfor deduplication similarity units that contain at least one inputchanged tracked block; representing in the similarity index only alatest generation of repository data, wherein a single latest entry ismaintained for a representative value in the similarity index; applyingimplicit deletion in the similarity index such that a similarly indexentry is not removed or overwritten until a change with associated dataof the similarity index entry is detected; maintaining a subset of bytesof the representative value in a similarity index entry to reduce aninput/output (I/O) load on the similarity index of the deduplicationsystem; deduplicating a respective one of the similarity units with acorresponding similarity unit of a previous snapshot; and examining adeduplication coverage; wherein when a deduplication coverage thresholdis not met, a similarity search is conducted and the respective one ofthe similarity units is deduplicated with a found similarity unitresiding in the similarity index.
 2. The method of claim 1, furtherincluding designating the similarity index entry for removal if at leastone of: snapshot data corresponding to the similarity index entry isdeleted, and new snapshot data is processed having new representativevalues of a new snapshot.
 3. The method of claim 2, further includingprocessing the similarity index entry for removal by at least one of:updating the new snapshot data, and identifying the similarity indexentry for removal within a search operation and removing the similarityindex entry for removal.
 4. The method of claim 3, further includingupdating the similarity index entry designated to be removed by a newsimilarity index entry that has a same representative value.
 5. Themethod of claim 1, further including configuring the similarity indexentry to include a storage reference to digests of a respective one ofthe similarity units to which the representative value corresponds to.6. The method of claim 5, further including storing remaining bytes ofthe representative value within the digests of the similarity unit.
 7. Asystem for reducing resource consumption of a similarity index in datadeduplication, comprising: a processor, operable to perform the datadeduplication, wherein the processor: partitions input snapshot datainto changed tracked blocks, groups the changed tracked blocks intoenclosing similarity units, processes for deduplication similarity unitsthat contain at least one input changed tracked block, represents in thesimilarity index only a latest generation of repository data, wherein asingle latest entry is maintained for a representative value in thesimilarity index, applies implicit deletion in the similarity index suchthat a similarly index entry is not removed or overwritten until achange with associated data of the similarity index entry is detected,maintains a subset of bytes of the representative value in a similarityindex entry to reduce an input/output (I/O) load on the similarity indexof the deduplication system, deduplicates a respective one of thesimilarity units with a corresponding similarity unit of a previoussnapshot, and examines a deduplication coverage; wherein when adeduplication coverage threshold is not met, a similarity search isconducted and the respective one of the similarity units is deduplicatedwith a found similarity unit residing in the similarity index.
 8. Thesystem of claim 7, wherein the processor designates the similarity indexentry for removal if at least one of: snapshot data corresponding to thesimilarity index entry is deleted, and new snapshot data is processedhaving new representative values of a new snapshot.
 9. The system ofclaim 8, wherein the processor processes the similarity index entry forremoval by at least one of: updating the new snapshot data, andidentifying the similarity index entry for removal within a searchoperation and removing the similarity index entry for removal.
 10. Thesystem of claim 9, wherein the processor updates the similarity indexentry designated to be removed by a new similarity index entry that hasa same representative value.
 11. The system of claim 7, wherein theprocessor configures the similarity index entry to include a storagereference to digests of a respective one of the similarity units towhich the representative value corresponds to.
 12. The system of claim11, wherein the processor stores remaining bytes of the representativevalue within the digests of the similarity unit.
 13. A computer programproduct for reducing resource consumption of a similarity index in datadeduplication by a processor, the computer program product comprising acomputer-readable storage medium having computer-readable program codeportions stored therein, the computer-readable program code portionscomprising: an executable portion that partitions input snapshot datainto changed tracked blocks; an executable portion that groups thechanged tracked blocks into enclosing similarity units; an executableportion that processes for deduplication similarity units that containat least one input changed tracked block; an executable portion thatrepresents in the similarity index only a latest generation ofrepository data, wherein a single latest entry is maintained for arepresentative value in the similarity index; an executable portion thatapplies implicit deletion in the similarity index such that a similarlyindex entry is not removed or overwritten until a change with associateddata of the similarity index entry is detected; an executable portionthat maintains a subset of bytes of the representative value in asimilarity index entry to reduce an input/output (I/O) load on thesimilarity index of the deduplication system; an executable portion thatdeduplicates a respective one of the similarity units with acorresponding similarity unit of a previous snapshot; and an executableportion that examines a deduplication coverage; wherein when adeduplication coverage threshold is not met, a similarity search isconducted and the respective one of the similarity units is deduplicatedwith a found similarity unit residing in the similarity index.
 14. Thecomputer program product of claim 13, further including an executableportion that designates the similarity index entry for removal if atleast one of: snapshot data corresponding to the similarity index entryis deleted, and new snapshot data is processed having new representativevalues of a new snapshot.
 15. The computer program product of claim 14,further including an executable portion that processes the similarityindex entry for removal by at least one of: updating the new snapshotdata, and identifying the similarity index entry for removal within asearch operation and removing the similarity index entry for removal.16. The computer program product of claim 15, further including anexecutable portion that updates the similarity index entry designated tobe removed by a new similarity index entry that has a samerepresentative value.
 17. The computer program product of claim 13,further including an executable portion that configures the similarityindex entry to include a storage reference to digests of a respectiveone of the similarity units to which the representative valuecorresponds to.
 18. The computer program product of claim 17, furtherincluding an executable portion that stores remaining bytes of therepresentative value within the digests of the similarity unit.